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ABSTRACT 

^ The misuse of evaluation is discussed. Boundaries of 

misutilization are not well understood, and distinctions between 
misutilization and non-use or abuse are not clear. In general, the 
responsibillity for evaluation misuse is with the user, while the ' 
responsibility for misevaluation is with the evaluator. A distinction 
.is also needed between intentional and uninten^'^^ional misuse. Misuse 
of evaluation may be further defined as misuse of commissioning the 
evaluation, misuse of the evaluation process, and misuse of the 
finding*^. A more precise definition will r^ejine the nature of 
misiitilization in order to estimate how often it occixrs. (SLD) 
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Considerable attention has been focusedMn recent years on the issue of evaluation 
utilization. Although only. a few were discussing utilization in the 197d's, it has become 
the "hot" topioof the late 1980*s. Indeed, last October, evaluation utilization was the 
theme of the most recent meeting of the American Evaluation Association in Boston. 

The attempts during the late 1S60*s at distinguishing between "evaluation" and 

"research" activities is, in part, responsibio for establishing ah ^^ropriate context for 

the consideration of evaluation! utilization. As Carol Weiss noted: 

The basic ~ ration ale for evaluation is that it provides information for 
action. Its primary justification is that ii contributes to the 
rationalization of decision making. ...unless it gains serious hearing when 
program decisions ar^e-m^, it fails in its major purpose (1966). 

Prior to the "action" emphasis in evaluation, there was very -little attontion to 

whether evaluation made a difference. The redefinition of evaluation as information for 

action or decision making highlighted the deficiencies of evaluation practice at that time. 

A variety of publications attested to the "underutilization" (if not downright 

impracticality and unimportance) of evaluation. For example, Rippey noted: - - 

At the moment there seems to be no indication that evaluation, although 
the law of the land, contributes anything to educational practice, other 
than headaches for ♦he researcher, threats for the innovators and 
depressing articles for journals devoted to evaluation (Rippey, 1973, 
pg.9). i 
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As a rasult of these concerns for the inadequacies of evaluation-, a number of 
researchers began the systematic study of conditions associated with the utilization of 
evaluation results. Studies of evaluation utilization (Patton, et.al, 1976; Alkin, Oaillak 
& White, 1979; Daillak,, Alkin, Stecher, 1981; Braskamp, Brown and Newman, 1978; 
King, Thompson and Pechman, 1982) have amplified the research domain and provided 
a multitude of insights regarding the conditions of evaluation use. Furthermore, a 
number of literature reviews have adequately summarized the factors that affect 
utilization and the 'research related to such -relationships. (See Leviton & Hughes, 
1981; Hansen, Martin & Oxford, 1981; and Alkin, 1985.) 

Evaluation Misuss 

While we know a great deal about how to increase the potential for utilization, we 
know very little about misuse and abuse of evaluation, and how to prevent it. The notion 
'r,of misutilization is certainly not a recent phenomenon, however. In 1973, Mushkin 
warned that "evaluators have .not sufficiently safeguarded their statements from 
misinterpretation" (p.34). In 1977, Cook and Pollard discussed four studies in which 
•aspects of misutilization were identified. Later, Weiss and Bucuvalas (1980) stated 
that "sometimes^decision makers misuse research" (p.11). More recently, in February 
1985, a group of- ten evaluation professionals attended an informal session at the "UCLA 
Malibu Conference Center." The session focusjed primarily on evaluation utilization, 
however, a portion of the dialogue that took place dup'ng the meeting touched upon the 
topic "of misutilization (see, Alkin, 1988). 

Despite the fact that there has been some talk about misutilization, its boundaries 
are not very well understood. Patton (1988) noted at the Malibu meeting, "we really 
don't knojj/ m.uch about the .dimensions of It [misutilization] or how it occurs" (p.165). 
Others at^th^t^eeting (Ross Conner, Ernest House, Mike Kean, Jean King, Susan Klein, 



Alex Law,,Milbrey McLaughlin, Carol Weiss and the first author of this paper) noted the 
need for further study. ' 

The purpose of this paper is to highlight some potential issues surrounding the - 
concept of misutilization and some distinctions 'betwee^n the term misuse (or 
misutilization) and other related concepts-such'^ as non-use and abuse. Potential 
examples of misutilization will be presented, and several categories or major types of 
misutilization will be proposed. * 
Defining MisiJtili2atinn 

First, it will be helpful to clarify, to. the extent possiblej the relationship 
between utilization, and misutilization. Patton (1988) contends that these two concepts 
are separate dimensions. "One dimension is a continuum' from nonutilization to 
utilization. A second continuum is nonmisutilization to misutilization" (p.336). Given 
this distinction, it follows that actions such as shelving evaluation reports might be 
•exemplary of non-use, but- not ne'cessarily misuse. 

What, then, is misutilization? This question is not easily answered. Even though 
researchers seem to agree that misuse of evaluation occurs, (e.g., Mushkin, 1973; Cook-^ 
and Pollard, 1977; Weiss and Bucuvalas, 1980;^atton, 1988), there does not yet exist 
a consensual definition of misutilization. Arrivin^at and justifying such a definition is 
not as simple as It may appear. , 

For example, consider a case in which an administrator blatantly squashes 
several negative evaluation reports to prevent the results from reaching the general 
public. On the surface, such an action appears to be a prime case of misutilization. Now, 
consider that same action (i.e., suppressing negative reports) in a situation where the 
reports were invalid due to poor data collection. Many researchers (e.g.. King, 1982) 
consider intentional non-use of poorly conducted evaluations as responsible and 
appropriate action. Thus, misutilization in one situation may be conceived of as 
appropriate non-use in another. 



One possible definition of misutiiization couid be the intentional (and even 
malicious) manipulation of some aspect of an evaluation (evaluative results, for 
example) in order to gain something--positioh or support, for instance. There are, of 
course, maiy other potential ways to define misuse. Before a consensual definition of' 
misutiiization can be reached, however, some value-based criteria need to be 
established. 

A related issue to the problem' bf defining misuse is that of determining who bears 
the responsibility for misuse. If Mushkin (1973) is correct in stating that ev 'uators 
have not taken the necessary steps to protect their statements from misinterpretation, 
are evaluators 'a\ fault for misuse? "Or, dcss the responsibility rest with users of 
evaluations? Perhaps, as Co^k and Pollard (1977) contend, both evaluators and users' 
contribute to the misutiiization of evaluations. 

. In some instances, seeming misuse is really attributable to the evaluator. A po'or 
report or failure to be sufficiently^ sensitive to users' information needs as well as their 
abilities to interpret data are two such examples. Additionally, the use of inappropriate 
methodologies, such as utilizing flawed data collection techniques or failing to Identify 
qualifications of findings, is reflective of bad practice (much like malpractice in the 
field of medicine). For clarity, we contend that malpractice by evaluators should be 
referred to as instances of misevaluation rather than misutiiization. Generally, when 
we allude to misuse, the focus is on users rather than evaluators because.ijs.ers are more 
attuned to the political ramifications of certain types of evaluations (and resiJits), and 
have a greater stake in how such results are perceived. Thus, users, we believe, arT" 
more likely than evaluators to intentionally manipulate some part of an evaluation (e.gj, 
the conclusions). 

Undoubtedly, there wilLbe borderline situations that might be viewed as either 
misuse or misevaluation (etg., when an evaluator succumbs to pressures to alter 
conclusions in a report). But, on the whole, wp define the responsibility for evaluation 



misuse as resting with users, while the responsibility for misevaluation lies with 
evaluators. 

p-l Another issue j;elated to misuse is the difference between intentional and 

^ unintentional misuse, and the role of each in the overall problem. Patton (at the Malibu 
meeting) addressed this issue briefly. He recognizes the distinction between \hese two 
types of misuse, and contended that unintentional misuse may^ be less of a problem 
because "unintentional misuse can be corrected through the processes aimed at 
increasing appropriate and propsr use; (p. 336). In short, educating potential users 
about the nature of evaluation and sensitizing them to appropriate modes of evaluation 
use can be beneficial. Alkin's A Guide for Evalijatinn Haniffiptj M ffFi ftrff (1985) is 
desigried to serve precisely that purpose. ; , 

Even though the distinction between "intentional" and "unintentional" ^misus-.i 
seems to be an important /one, not ali researchers have made such ■■=i separation. The 
views of Cook and Pollard (1977) are illustrative of this point. These researchers 
contend that publicly ascribing findings that differ from those in a fin', evaluation 
report is a type of misutilization. This is an easily accepted point of view. On the other 
hand. Cook and Pollard also believe that "mirutilization is involved when the findings in 
the report are accepted uncritically, though nearly all competent methodo legists would 
have ascribed , ss validity to them than the users did" (p.161). 

These two examples of misuse seem to us to differ considerably. In our view, 

- lumping the two 'l^inds of misuse in a single category is questionable. In the first case, 
the misusers are more likely to be cognizant of the inappropriateness of tlieir actions, 
given that they have a written document against wliich to compare their public 

•statements. This instance of intentional evaluation misuse we ^would refer to us 
evaluation abuse. In the ^^econd case, however, the misusers could easily be acting out of 
ignorance. Accusing users who lack methodological sophistication of misuse' is debatable. 
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One might even contend that misuse resulting from users' lacl< of knowledge is, in 
v^ctuality, a type of mi?3valuation--as we previously noted. / 

The various relationships just discussed are summarized in Figure 1 below. Note 
that we have distinguished between properly done and poorly done evaluations, between 
use and non-use, the users' intents, and the extent to which users are informed. This 
leads to a classification system yielding situations of use, non-use, misuse, abuse, and 
misevaluation. 

Figure 1 

EVALUATION USE RELATIONSHIPS: SOME PREUMINARY THOUGHTS 



: ' USE " 
WELL DONE 

\ 

NON-USE- 



EVALUATION 



POORLY DONE 




\ 

NON-USE 



/UNINTENTIONAL ' — NON-USF 

-PURPOSEFUL - MISUSE 

7^ INTENTIONAL/BLATANT — ABUSE 

i:»"FORMEDUSER MISUSE 
UNINFORMED USER - MISEVALUATION 

- JU STIFIE D 
NON-USE 



• Further ExamplPs from th P Literati irfi 

Our examination of the evaluation .utilization literature presents numerous 
examples of situations which help to define '"misuse" and/.associated terms. Two reputed 
instances of .nisuse that are found in the literature we believe are instead Instances of 
misevaluation. These are:- 1) when the evaluator fa-fs-to communicate results to- all 



users or 2) when the evaluator succumbs to bureaucratic pressures to produce certain 
kindS; of results. 

To facilitate further thinking on the topic of misutilization, it may be helpful to 
identify major categories of misuse based on cases of misuse cited in the literature thus" 
far. Currently, it seems plai/sible to separate misuse into three categories: 

(1) misuse <of commissioning an evaluation, 

(2) misuse of the evaluation process, and 

(3) misuse of evaluative findings. 

Basically, these categories reflect the general stages of evaluation during which 
misutilization may occur. Figure 2 categorizes examples of misut'lization based on the 
above scheme. Note that the^ may be instances of misuse or abuse depending upon the 
intent of the user. Differences associated with user-intent were depicved in the prior 
Figure 1. 

The difficulties in classifying instances as either non-use, misuse, abuse, etc. 
are. demonstrated in Figure 3, whiqh presents several borderline cases. The first two of 
these we would consider as questionable .cases of misuse. If the action taken was due to 
user lack of competence the evaluator may share some of the fault. The third example we 
would consider a possible instance of misuse depending ypon the intent of the user and 
the extent to which the selected evidence (or verbage) is non-representative of the 
report as a whole. Likewise, the fourth and fifth examples are dependent upon user 
intent and the quality of the evaluation. One could hardly call the non-use in number 
five "misuse" if the quality of the evaluation was poor. 
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Figure 2 

■ CATEGORIES OF MISUSE/ABUSE * 

A Misuse of Commissioning the Evaluatldn 

0 Commissioning evaluation for symbolic reasons (e.g., for political gain ' 
only, for publicity onjy, to gain funding only) (Weiss,1973) 

B. Misuse of the Evaluation Process 

0 Using evaluations to delay action (Weiss, 1973) 

0 Using evaluations to avoid taking responsibility (Weiss, j973) 

C Misuse of the Evaluation Findings 

; 0 Changing the.,wording of . Walualive conclusions (House 1988) 

0 Selectively reporting results (House, 1988; Weiss & Bucuvalas 
1980) 

0 Blatant non-use (of sound evaluations) (Patton, 1988) 

•0 Ascribing findings to a study that differ from the actual results (Cook 
& Pollard, 1977) ' 

0 Inaccurate transmission of results (adapted from Weiss, 1 972) 

0 'Oversimplifying findings (Weiss & Bucuvalas; 1980) 

0 Dispensing with essential qualifications (Weiss & Bucuvalas, 1980) 



* Depending upon the intent of the user. 



ERIC 



8 



Figure 3 

BORDERLINE CASES OF MISUTIUZATION 



1 . Accepting findings uncritically, even though nearly all "competenf 
methodologists would have ascribed less validity to the findings than 
would users. 

2. Using results from methodologically flawed studiss. 

3. Selecting paragraphs or sentences from results of an evaluation without 
representing the entire picture. 

4. Releasing results prematurely. 

5. Shelving the evaluation findings to reduce the likelihood of utilization of 
results. 

Next Steps 

Clearly, there are many unanswered questions regarding the concept of 
misutilization. This paper has touched but a few of the complexities surrounding this 
problem. The topic of misutilization has not been systematically explored. Clearly, a 
more precise definition is required. 

One approach to studying misutilization might be to create a list of potential 
examples of misuoe (some based on the literature and others contributed) and examine 
the extent of consensus among prominent evaluators. Such data could be supplemented 
with the misutilization-related perceptions of the people in the field (i.e., u.sers and 
other practitioners). Collectively, this type of information would enable us to begin to 
refine the nature of misutilization and to estimate the extent of Its occurrence. 
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